This invention relates to a technology for recovering a failure in a stream data processing system.
Up to now, a database management system (hereinafter, referred to as “DBMS”) plays a dominant role in data management for a corporate information system. In the DBMS, data to be processed is stored in a storage system, and a highly-reliable processing typified by a transaction processing is realized for the stored data.
In contrast, in recent years, there is an increasing demand for a data processing system which performs a real-time processing on a large amount of data arriving from moment to moment. For example, in a case of a financial application for assisting stock trading, it is one of the most important challenges for the system how to quickly respond to stock price fluctuations. With a system such as the conventional DBMS for temporarily storing stock data in a storage system and then retrieving the stored data, there is a fear that the speed of the stock price fluctuations cannot be coped with.
Therefore, a stream data processing system is proposed as a data processing system suitable for a real-time data processing as described above. For example, a stream data processing system called “STREAM” is disclosed in R. Motwani, J. Widom, A. Arasu, B. Babcock, S. Babu, M. Datar, G. Manku, C. Olston, J. Rosenstein and R. Varma: “Query Processing, Resource Management, and Approximation in a Data Stream Management System”, In Proc. of the 2003 Conf. on Innovative Data Systems Research (CIDR), January 2003.
In the stream data processing system, unlike the conventional DBMS, queries are previously registered in the system, and the registered queries are continuously executed in response to the arrival of the data. The term “stream data” used herein is not a large item of logically-continuous data such as a video stream, but represents a large amount of time-series data of relatively small items that are logically independent of each other including stock-price feed data for a financial application, POS data for a retail trade, probe car data for a traffic information system, an error log for computer system management, and sensing data generated from a ubiquitous device such as a sensor or an RFID. To process the stream data in real time is called “stream processing”.
The stream data keeps arriving at the system, thereby making it impossible to perform the processing in real time if the processing is started after waiting until the end of the arrival. In addition, data items that have arrived at the system must be processed in arrival order without being affected by the load on the data processing.
The system “STREAM” has introduced a notion called a sliding window (hereinafter, referred to simply as “window”) for realizing a real-time processing by cutting out a part of the stream data items that are arriving at the system by specifying the width of a time period such as latest 10 minutes or the width of the number of items such as latest 1,000 items.
As a preferable example of a description language for queries including such a window specification, Continuous Query Language (CQL) is disclosed in above-described “Query Processing, Resource Management, and Approximation in a Data Stream Management System”. CQL has been developed by extending Structured Query Language (SQL), which is widely used for the DBMS, so as to be capable of the window specification.
FIG. 22 is a diagram illustrating an example of CQL. A query 2201 illustrated in FIG. 22 is an example of the query described in above-described “Query Processing, Resource Management, and Approximation in a Data Stream Management System”.
The query 2201 is a query for calculating a total count of access that has been made to a certain web proxy server from the domain “stanford.edu” for one day. “Requests” represents web access data that has keep arriving at the web proxy server, and is not quiescent data such as a table to be handled in the DBMS but continuous stream data. Therefore, the total count of the access cannot be calculated without specifying which part of the stream data to be a subject as in the window specification “[Range 1 Day Preceding]”.
The stream data cut out by the window is retained in the memory and used for a query processing. As typical window specification methods, there are a “Range” window (hereinafter, referred to as “time window”) for specifying the width of the window by the time period and a “Row” window (hereinafter, referred to as “row window”) for specifying the width of the window by the number of data items. For example, if the time window is set to 10 minutes, data items for the latest 10 minutes are to be subjects of the query processing, and if the row window is set to the number of 10, the latest 10 data items are to be subjects of the query processing.
Meanwhile, as a method of recovering the stream data processing system, Hwang, J.-H. Balazinska, M. Rasin, A. Cetintemel, U. Stonebraker, M. Zdonik, S.: “High-Availability Algorithms for Distributed Stream Processing” Proceedings of the 21st International Conference on Data Engineering, (ICDE2005) April 2005 discloses an ActiveStandby scheme in which an active system and a standby system are caused to operate even in a normal mode and a PassiveStandby scheme in which memory is periodically copied from the active system to the standby system.
In the ActiveStandby scheme, data is transmitted from a data transmission application to both the active system and the standby system, which both perform the stream data processing. Then, in a result reception application, a check is performed on consistency between output stream data output from the active system and output stream data output from the standby system, and only the output stream data output from one of the systems is handled. Even if a failure occurs in the active system, the standby system is constantly operating, and hence a stream processing system can be continued by switching to the standby system.
In the PassiveStandby scheme, an input stream is subjected to the stream processing in the active system, while the input stream data is buffered without being subjected to the stream processing in the standby system, and processing results in the active system are periodically copied to the standby system. If a failure occurs in the active system, the stream processing system can be continued by processing the input stream data starting from the memory that has been copied to the standby system.